MINIMISING THE TIME TO REACH A TARGET AND RETURN 



SAUL JACKA 



Abstract. Motivated by a problem in Metropolis-coupled, Markov chain Monte 
Carlo, we seek to minimise, in a suitable sense, the time it takes a (regular) diffusion 
with instantaneous reflection at and 1 to travel from the origin to 1 and then return. 
The control mechanism is that we are allowed to chose the diffusion's drift at each 
point in [0,1]. We consider the static and dynamic versions of this problem, where, 
in the dynamic version, we are only able to choose the drift at each point at the 
time of first visiting that point. 



July 12, 2012 

1. Introduction and problem motivation 

1.1. Introduction. Suppose that is tlie diffusion on [0, 1] started at and given 
by 

dXf = a{Xt^)dBt + fx{Xt^)dt on (0,1) 

with instantaneous reffection at and 1 (see [H] or [B] for details). Where there is no 
risk of confusion we omit the superscript fi. 

Formally, we define to be the first time that the diffusion reaches x, then we define 
S = S{X), the shuttle time (between and 1), by 

S{X) = mi{t > Ti(X) : Xt = 0}. 

In this article, we consider the following problem (and several variants and general- 
izations) : 

Problem 1.1. Minimise the expected shuttle time KS ; i.e. find 

infE[5(X'^)], 

where the infimum is taken over a suitably large class of drifts fj,, to be specified in 
more detail later. 



Given the symmetry of the problem it is tempting to hypothesise that the optimal 
choice of fi is 0. We shall soon see that, in general, this is false, although it is true 
when (T = 1 

We actually want to try and minimise S (and additive functionals of X^ evaluated at 
S) in as general a way as possible so we extend Problem 11.11 in the following ways: 
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Problem 1.2. Find 

s 



infE[/ fiXndt] 



for suitable positive functions f ; 
and 

Problem 1.3. Find 



supE[exp(- [ a{Xt^)dt)]; 

M Jo 

for suitable positive functions a. 

We shall also solve the corresponding discrete statespace problems (in both discrete 
and continuous time). 

Although we will prove more general versions it seems appropriate to give a prelimi- 
nary statement of results in this context. 

Theorem 1.4. Suppose that a > 0, that f is a non-negative Borel-measurable function 
on [0, 1] and that 

(1.1) ^GLi([0,l],A), 

a 

where A denotes Lebesgue measure, then 

fS 



inf E[[ f{X^)dt] = {[ \r-^duf. 

measurable A* Jg Jo \ O' [U) 



If, in addition, \J ^ is continuously differentiate and strictly positive on (0,1), then 
the optimal drift is fi given by 

Theorem 1.5. Suppose that a > 0, that a is a non-negative Borel-measurable func- 
tion on [0, 1] and that 



(1.2) ^EL\[0,l],\), 

(7 



sup E[exp(- /" a(X^)rft)] = cosh"^( /" W^^p^c/m). 

measurable fi JQ J Q \i ^ \^) 



If, in addition, continuously differentiable and strictly positive on (0,1), then 

the optimal drift is fi given by 

We will eventually solve the problems dynamically, i.e. we will solve the corresponding 
stochastic control problems. However, we shall need to be careful about what these are 
as the problem is essentially non-Markovian. Normally in stochastic control problems, 
one can choose the drift of a controlled diffusion at each time point but this is not 
appropriate here. In this context, it is appropriate that the drift is 'frozen' once we 
have had to choose it for the first time. We shall formally model this in section H] . 
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1.2. Problem motivation. The problem models one arising in Metropolis-coupled, 
Markov chain, Monte Carlo (MCMCMC or MC^) simulation. Essentially the level 
corresponds to the temperature in a "heat bath". The idea is that when simulating 
a draw from a highly multimodal distribution we use a reversible Markov Process 
to move between low and high temperature states (and thus smear out the modes 
temporarily) so that the Markov chain can then move around the statespace, then at 
low temperature we sample from the true distribution (see [2]). 



2. Notation and some general formulae 



Definition 2.1. Denote by the standardised scale function for and by m'^ the 

corresponding speed measure. Since X is regular and reflection is instantaneous we 
have: 



X 



s>'{x)= I exp(-2^ ^^dt^du, 







r du r^^-^{^i:^/t) 

m'^ix) = 2 / = 2 / ^-du, 



(see W- 



From now on, we shall consider the more general case where we only know that (drop- 
ping the /i dependence) s and m are absolutely continuous with respect to Lebesgue 
measure so that, denoting the respective Radon-Nikodym derivatives by s' and m' we 
have 

2 

s'm' = — Lebegue a.e. 

For such a pair we shall denote the corresponding diffusion by X*. We underline that 
we are only considering regular diffusions with "martingale part" J adB or, more 
precisely, diffusions X with scale functions s such that 

ds{Xt) = s'{Xt)a{Xt)dBt, 

so that, for example, sticky points are excluded (see [4j| for a description of driftless 
sticky BM and its construction, see also [3] for other problems with solutions of 
stocahstic differential equations ). Note that our assumptions do allow generalised 
drift: if s is the difference between two convex functions (which we will not necessarily 
assume) then 



(2.1) 



Xt = x+ I a{X:)dB.^-\ I L»(X)^ 
Jo Jr 



s"{da) 



where s'_ denotes the left-hand derivative of s and s" denotes the signed measure 
induced by s'_. 

Definition 2.2. For each y G [0, 1], we denote by (py the function 

(t)y: x^E^l fiXs)ds], 
Jo 

where, as is usual, the subscript x denotes the initial value of X under the correspond- 
ing law Fx- 



4 SAUL JACKA 

Theorem 2.3. For < x < y , (py is given by 



(2.2) 4'y{x) = 4>y{x) =W / f{u)m'{u)s{v)dudv, 

Jx Ju=0 

while for < y < x (py, is given by 

(2.3) ^y{x) = [ [ f{u)m'{u)s'{v)dudv. 

J y J u=V 

In particular, 

rS pi rl 

(2.4) Eo[/ f{X^')dt= / / f{u)m'(u)s'{v)dudv. 

Jo Jo Jo 

Proof: This foUows immediately from Proposition Vn.3.10 of [8] on observing that, 
with instantaneous reflection at the boundaries, the speed measure is continuous. □ 

We give similar formulae for the discounted problem: 
Definition 2.4. We denote by ipy the function 



ipy : a; Ea;[exp(- / a{Xs)ds)]. 
Jo 



Theorem 2.5. (i) Either 

(2.5) / a{u)dm{u) < oo, 

Jo 

or 



Eo[exp(- / a{Xs)ds) = 0, 
Jo 



in which case 



/ a{Xs)ds = oo a.s. 
Jo 

(ii) Now suppose that 1^2. 5\} holds. For each n, denote by In{x) the integral 
In{x) j a{ui) . . . a{un)dm{ui) . . . dm{un)ds{vi) . . . ds{vn) 

0<U\<.V\<.U2 ■ ■ ■<Vn<X 

and by In{x) the integral 
/„(x) =^ J a{ui) . . . a{un)dra{ui) . . . dni{un)ds{vi) . . . ds{vn) 

X<V\<U\<.V2---<:Un<:l 

with Jo = /q = 1- Now define G and G by 

oo oo 

(2.6) G{x) = ^n{x) and G{x) = ^ /„(x). 



The the sums in Ii2.6\) are convergent and for x < y 
while for x > y 
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Proof: 

(i) Note first that < oo follows from regularity. 

We consider the case where x < y. Now suppose that a is bounded. It 
follows that tjjy > for each y since J^^ a{Xs)ds is a.s. finite for bounded 

a. Now, setting = exp{— J^^"^^ a{Xs)ds)ilj{Xt/^Ty), it is clear that is a 
continuous martingale. Thus, writing 



ilj{Xt;,Ty) = exp( / a{X,)ds)Nt 
Jo 

it follows that 

PtATy Pt PUATy 

i^iXtATy) - / aip{Xu)du = / exp( / a{Xs)ds)dNu, 



Jo Jo Jo 

and hence is a martingale. Thus we conclude that is in the domain of^^, 
the extended generator for the stopped diffusion X'^y, and 

A^ipy = aip. 

Since the speed and scale measures for X and coincide on [0, y] and using 
the fact that ip'{0) = 0, we conclude from Theorem VII. 3. 12 of [8] that 

(2.7) i^yi^) = "^2/(0) + / / s {v)a{u)il)y{u)ni {u)dudv for x < y. 

Jv=0 Ju=0 

A similar argument establishes that 

(2.8) i'yix) = i'yi^) + / / s {v)a{u)il>y{u)m! {u)dudv for x > y. 



v=x J u=v 



Now either 

min(^i(0),^o(l)) =0, 

in which case 

Eo[exp(- / aiXs)ds) = = 0, 

Jo 

or 

(2.9) min(V'i(0),^o(l)) = c>0. 

Suppose that (12.91) holds, then (since ipi is increasing) it follows from (12.71) 
that 

pi PV 

(2.10) V^i(l-) > c+ / s'{v)ca{u)m'{u)dudv 

Jv=0 Ju=0 

= c(l + / / s' {v)a{u)m' {u)dudv) 

J u=0 J v=u 

> c(l + (s(l) - s(^)) f\{u)m'{u)du). 
Similarly, we deduce that 



1 

V'o(0+) > c(l + s(-) / a{u)m\u)du). 
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Thus, if fl2.5p fails, (I2.9p cannot hold (since if (12 .Sp fails then at least one 
of a{u)m'{u)du) and jl a{u)m'{u)du) is infinite) and so we must have 

To deal with unbounded a, take a monotone, positive sequence «„ increasing 
to a and take limits, 
(ii) Suppose now that (I2.5p holds. Setting 

G(x) = ^ 

we see that G satisfies equation (12. 7p with G{0) = 1. 

Convergence of the series ^ In and ^ /„ follows from the following bounds 
on In and /„. 

Lemma 2.6. Let 

B{y) *= / a{u)dm{u) and B{y) =^ /" a{u)dm{u)^ 

then 







(2.11) U.) < (-Wf 'f »" .nd /.(.) < '-^(-^>^(-^» 



(n!)2 - (n!)2 ' 

■s(x) =^ s(l) — s{x). 

Now we establish the first inequality in (12. lip by induction. The initial 
inequality is trivially satisfied. It is obvious from the definition that 

In+lix) = / a{u)In{u)dm{u)ds{v), 

Jv=0 Ju=0 

and so assuming that /„(■) < ^^^'^n,^^ 

X V 

(2.12) In+i{x) < [ [ a{u)^^^^P^dm{u)ds{v) 



/f Biu]^ 
I aiu) dm{u)s{v)^ds{v) since s is increasing 

J [niy 



v=0 u=0 

X V 



-s{vYds{v) 



n!(n + 1)! 

BixY+^ 



v=0 

X 



< — r- / s(vYds(v) since B is increasing 

~ n\{n + 1)1 J ^ ' ^ ' 



v=0 



(s(x)5(x))"+i 
((n+l)!)2 ' 

establishing the inductive step. A similar argument establishes the second 
inequality. 
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Now by iterating equation (12.71) we obtain 



n-l 



G{x) = 



k=0 

+ J a{ui) . . . a{un)G{un)dm{ui) . . . dm{un)ds{vi) . . . ds{vn 

0<U\<.V\<.U2 ■ ■ ■<Vn<X 

Since G is bounded by -^jj^ we see that 



0<G(x)-5^4(x)<-^4(x). 



k=0 

A similar argument establishes that 

n-l 



0<G(x)-^4(x)<-i-4(x) 



^0 

and so we obtain (12.61) by taking limits as n — t- cxd. 



□ 



3. Preliminary results 

For now we will state and prove more general, but still non-dynamic versions of 
Theorems 11.41 and II. 5[ 

We define our constrained control set as follows: 

Definition 3.1. given a scale function sq ~ A and C, a Borel subset of [0,1], we 
define the constrained control set Ai'^' by 



(3.1) -^so ~ {scale functions s : ds\c = dso\c (^nd s ~ A} 

The corresponding controlled diffusion X* has scale function s and speed measure m 
given by 

, 2 

Theorem 3.2. For any scale function s ~ A, define the measure V by 



/(") 



and the measure J by 



'2/(n) 



then, given a scale function so. 



inf Eo[ / fiX^)dt] = (v/so(C)Po(C) + J(C^))^ 
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The optimal choice of s is given by 

So (fix) : on C 



s{dx) 



Y^y^dx -.onC^ 



Proof: Note first that, from Tfieorem 12.31 

(3.2) Eo[J f{Xt)dt] = 0i(O) + 0o(l)= j j f{u)s{dv)m{du) 

= SoiC)r''{C) + J [r''{C)s{dv) + s{C)f{v)m{dv)] 

+ - J J [f {u) s{dv)m{du) + f {v)m{dv) s{du)\. 

Note tliat in tlie last term in (13. 2p tlie factor ^ follows from the fact that we have 
symmetrised the integrand. Now, for s G M^^, we can rewrite (12. 4 p as 

(3.3) Eo[ f f{X,)dt] = So{C)F°{C) + I [r\C)s'{v) + s,{C) ^/[""l A dv 

Jo J Cr [V)S [V) 

2 J J ^a^{u) s'{u) a^{v) s'{v)^ 
We now utilise the very elementary fact that for a,b > 0, 

(3.4) inf [ax H — 1 = 2ab and if a, 6 > this is attained at x = \ —. 
x>o' X V a 

Applying this to the third term on the right-hand-side of (13. 3p we see from (13. 4p that 
it is bounded below by / / [J ^ir^flxdudv = (/^, \ y^^^duY = J^^C) and this is 

attained when s'(x) is a constant multiple of \J^^ a.e. on C^. 

Turning to the second term in (13. 3p we see from (13. 4p that it is bounded below 
by 2 v^so {C)P^ (C) sj^^dv = 2^so{C)P°{C)J{C') and this is attained when 

Thus, we see that the infimum of the RHS of (13. 3p is attained by setting s'{x) equal 
to ^ /"o^(g) ■sj on and this gives the stated value for the infimum. □ 

In the exponential case we only deal with constraints on s on [0,y]. 
Theorem 3.3. Assume that Ii2.5\) holds and define 



a{x) ' 

(i) Let G be as in equation Ii3.3\) . so that (at least formally) 
{ah' [^] '-2G) = \a^G" + /xG" - aG = 
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and let G* satisfy the '^adjoint equation" 

1 JaVG*']' 
-a(^^^-2G*)=0 

with boundary conditions G'*(0) = 1 and G'*'(0) = 0, so that 

(3.5) 

G*(x) = l+ f f '^^^^$^^s'iu)dudv=l + I I a(v)G*(v)dm(v)ds(u), 

Jv=oJu=0 (T^{V)S'{V) Jv=oJu=0 

then G* is given by 
(3.6) G*{x) = J2C 



oo 
n=0 

where 



(3.7) /*(x) =^ J a{vi) . . . a{vn)ds{ui) . . . ds{un)dm(vi) . . . dm{vn)- 

0<Ul<Vl<...Vn<:X 

(ii) r/ie optimal payoff for Problem \1.3\ is given by 



sup Eo[exp(- / a{X[)dt)] = ip{y) 



where 



^(y) = (VGG* cosh F{y) + V a^G'G*' sinh F{y)y 

with 



^ ^/2du / 2a 



The payoff is attained by setting d-{x)s'{x) = \J §^(l/) for all x > y (ify = 0, 
any constant value for a{x)s'{x) will do). 



Proof: 

(i) This is proved in the same way as equation fl2.6p in Theorem 12. 5[ 

(ii) First we define 

(3.8) -^ri(^) ^ J tt('^'i) . . . a{vn)ds{ui) . . . ds{un)dm{vi) . . . dm{vn)] 
and 

oo 

(3.9) G*{x)'^Y.^:{x). 

To prove (ii) we use the following representations (which the reader may easily 
verify) : 

n n 

(3.10) 4(1) = ^ UyK-miy) - ^\y) E UvWl-Jiv). 

m=0 m=l 
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and 

n n 

(3.11) 4(0) = J2 im{y)i:-miy) - ^'(y) E UyWLJ'iy) 

m=0 m=l 

It follows from these equations that 

(3.12) Eo [ / exp{-a{Xt)dt)] 

Jo 

= [G{y)G*iy) - a\y)G'{y){G*ny)] [G{y)G*{y) - a\y)G'{y){G*ny)]. 

Now essentially the same argument as in the proof of Theorem 13.21 will work. 
Multiplying out the expression on the RHS of fl3.12p we obtain the sum of the 

(a) iG(t/)G*(y) E [in{y)i*miy) + Uymy)] 

m>0,n>0 

(h) '^G'iy)lG*ny) E [I'niymny) + I'MiPJiy)] ; ^nd 

m>0,n>0 

(c) E [G(i/)(G*)'(y)J:(i/)/;„(i/) + G'(i/)G*(i/)(a'(y)/„(i/)], 

m>l,7i>0 

where in the first two terms we have symmetrised the sums. 
Using fl2.6p . (c) becomes 

y: I [G(y)((5*)'(,)4j^"^^---''("")''(^^)---''^^-) 



m> 



t'{Vi) . . .t\Vn)t'{Wi) . ..t'{Wm) 

where 

Dm,n{^) = {{u,v,w,z) eW" xW xR"" xR""-^ : x < Ui < vi < . . .Vn < I; 

and X < Wi < zi < . . . Wm < 1}, 

t is the measure with Radon-Nikodym derivative t' = as', and A denotes the 
measure with Radon-Nikodym derivative 4. Clearly each term in the sum is 

minimised by taking t' constant and equal to •y/p^^T^ b) !]■ 

The first two terms, (a) and (b), are each minimised by taking t' constant 
a.e. on [y, 1]. Substituting this value back in we obtain the result. 



□ 

Remark 3.4. We see that, in general, in both Theorems \2.3\ and \2.(A the optimal 
scale function has a discontinuous derivative. In the case where G = [0, y) there is 
a discontinuity in s' at y. This will correspond to partial reflection at y (as in skew 
Brownian motion- see |8] or [6]j and will give rise to a singular drift - at least at y. 

Remark 3.5. We may easily extend Theorems \2.3\ and \2.6\ to the cases where f 

or a disappears on some of [0,1]. In the case where N {x : f{x) = 0} is non- 
empty, observe first that the cost functional does not depend on the amount of time the 
diffusion spends in N so that every value for ds\N which leaves the diffusion recurrent 
will give the same expected cost. If X{N) = 1 then the problem is trivial, otherwise 
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define the statespace S = [0,1 — A(A^)] and solve the problem on this revised interval 
with the cost function f{x) == f{g^^{x)) where 

g:t^ A([0,t] niV") 

and 

g'^ : X I— !■ inf{t : g(t) = x}. 

This gives us a diffusion and scale function which minimises the cost functional 
on S. Then we can extend this to a solution of the original problem by taking 

ds = dsolN + dslN':, 

where dso is any finite measure equivalent to A and ds is the Lebesgue-Stiltjes measure 
given by 

~s{[0,t]) = s^{[0,X{[0,t]n N'^) = s^{g{t)). 
An exactly analagous method will work in the discounted problem 

4. The dynamic control problems 
We now turn to the dynamic versions of Problems 11.21 and 11.31 

A moment's consideration shows that it is not appropriate to model the dynamic ver- 
sion of the problem by allowing the drift to be chosen adaptively. If we were permitted 
to do this then we could choose a very large positive drift until the diffusion reaches 
1 and then a very large negative drift to force it back down to 0. The corresponding 
optimal payoffs for Problems 11.21 and 11.31 would be and 1 respectively. We choose, 
instead, to consider the problem where the drift may be chosen dynamically at each 
level, but only when the diffusion first reaches that level. Formally, reverting to the 
finite drift setup, we are allowed to choose controls from the collection M of adapted 
processes fi with the constraint that 

(4.1) flt=f^Tx,, 

or continuing the generalised setup, to choose scale measures dynamically, in such a 
way that s'{Xt) is adapted. 

Although these are very non-standard control problems we are able to solve them - 
mainly because we can exhibit an explicit solution, to whit, following the same control 
as in the "static" case. 

Remark 4.1. Note that this last statement would not be true if our constraint was 
not on the set [0,y]. To see this, consider the case where our constraint is on the set 
[y, 1] . If the controlled diffusion starts at x > then there is a positive probability that 
it will not reach zero before hitting 1, in which case the drift will not have been chosen 
at levels below Iti, the infimum on [0,Ti]. Consequently, on the way down we can set 
the drift to be very large and negative below Thus the optimal dynamic control will 
achieve a strictly lower payoff than the optimal static one in this case. We do not 
pursue this problem further here but intend to do so in a sequel. 

We need to define the set of admissible controls quite carefully and two approaches 
suggest themselves: the first is to restrict controls to choosing a drift with the property 
(14. ip whilst the second is to allow suitable random scale functions. 



there 
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Both approaches have their drawbacks: in the first case we know from Remark 13.41 
to expect that, in general, the optimal control will not be in this class, whilst, in the 
second, it is not clear how large a class of random scale functions will be appropri- 
ate. In the interests of ensuring that an optimal control exists, we adopt the second 
approach. From now on, we fix the Brownian Motion B on the filtered probability 
space (J't)t>o,IP). 

Definition 4.2. By an equivalent random scale function we simply mean a random, 
finite Borel measure on [0, 1] a.s. equivalent to Lebesgue measure and we define 

{A.2)Ai '= {equivalent random scale functions s : 

exists a martingale with = s' o s^^.a o s^^{Yy)dBy}. 

Jo 

We define the corresponding controlled process X'"^ by 

Xt = s-\Y,). 

For any sq E Ai we then define the constrained control set by 
(4.3) M'^' = {seM: ds\io,y) = c?So|[o,y)} 

Remark 4.3. Note that Ai contains all deterministic equivalent scale functions. An 
example of a random element of M. when a = 1 is s, given by 

(is|[oi) = d\] ds{x)\n]^ = lrfAl(Ti(B)<i) + exp(-2(x - ^))ciAl(Ti(B)>i), 

corresponding to X* having drift 1 above level | if and only if it reaches that level 
before time 1. 

Theorem 4.4. For each s E A4 let M/ denote the running maximum of the controlled 
process X^. Then for each Sq E Ai, the optimal payoff (or Bellman) process V'"^*^ 
defined by 



pS 

= essinf E[ / f{X^)dt\J^t 



is given by 
(4.4) 

def j £ f{X^^^)du + 2(v/so(Mf )Po(Mf ) + J{Mt°)Y - </>x:o(0) : /or < 1 
£ /(X:o)dn + 0o(Xr) : for M^^ = 1, 

( where (j) is formally given by equations ^2.2) and ^2.3\) with s = Sq), and the optimal 
control is to take 



t 

t — ""t — \ pt 



(4.5) /(,) = i4^4M/„,.,>Af» 

Proof: Consider the candidate Bellman process Vt- Using the fact that 

(4.6) N,^^ j\x:-)du-<P^so{Q) 
and 

(4.7) N','^MXn+ f f{X:°)du 

Jo 
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are martingales, is a continuous, increasing process, and 0i(O)+0o(l) = 2so(l)/''''(l) 
(so that V is continuous at Ti^X'^^)): 



dvt = 4( v/so(Mf)po(Mr) + J(Mr) 



"1 ^ Po(M''^'^) 1 



^ 2/((Mf': 

a2((Mf>) 



/(MP 

S(0 



Now, since Sq, and J are no n- negative it follows from fl3.4p that 

dvt > dNt, 



where 

with equality if 
(4.8) 



dNt = dNtlM^o^^ + dN'tlM^o^i, 



s\Mt 



po(Mf )(T2(Mi"°' 

Then the usual submartingale argument (together with the fact that v is bounded by 
assumption fll.ip ) gives us (14.41) . 

It is easy to check that s given by (14. 8 p is in J^'^^^sq. The fact that the optimal choice 

of s satisfies (14. 5 p follows on substituting s'x) = \J i^f^^y^j^ i^i the formulae for s and 
P and observing that the ratio jj^t^ is then constant on [y, 1]. □ 



Theorem 4.5. The Bellman process for Problem \1.3\ is given by 
l^/"= esssup E[exp(- / a{X^°)dt)\Tt] = Wt = exp{- a(Xf )rfs) V(Xf , Mj""), 

where 



G{x)'tp{y) ify<l, 



G{x) 



and 



^(y) = {\/Gd*coshF{y) + Va^G'G*'smhF{y)y 
(as in Theorem \3.3\ (ii)) with 
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The payoff' is attained by setting 



(4.9) 



aixjs [x] 



— ^(M;°) for all X > Mfo 
G'G* — 



Proof: The proof is very similar to that of Theorem 14.41 Note that ijj is continuous 
at the point (1, 1). 

For a suitable bounded martingale n, 



exp(- / «(Xf)rfs)G(Xf)V^'(M,^°)l(A,.o<i) + rfn, 



-2exp(- / a{X'^°)ds)G{X^"){V GG* coshF{Mt'°) + V a^G'G*' smhF{Mt')) 



X 



GG*y - V G'G*'] coshF(Mf') + {Va^G'G*')'] - Va^GG* sinhF(M4'o)] + dut 



Now 



2 

with equality attained when 



GG*y = Y ^ + ^ V "G - ^^^^*' ^^^""S (IMI), 



(4.10) 

Similarly, setting dm' 



VG'G*' = V^Gd*. 



a dm 

a ' 



{Va^G'G*')' 



'dG dG* 
ds dm^ 



1 d'G 
-m . 

2 dm°'ds 



dG* 1 A'^n* 



\ ds 



+ 7:3 



2 dsdra^ 



dG 
ds 
dG* 
dm°' 



If I d^G 
2\a'^s' dm"ds 

1 / 1 



dm°' _|_ 



dG 
ds 



dsdm^ 



dG 
ds 

dG* 
dm" 



2\a's 



-G^ 



dG* 
dm"' 
dG 
ds 



+ s'G* 



\ 



dG 
ds 

dG* 
dm" 



> 



with equality when 
(4.11) 



GG* 



a 



G 



dG* 
dm" 
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Now we can easily see (by writing ^ = ^G' and = a'^s'G*') that ( 14. lip implies 
f l4.10p so the standard supermartingale argument establishes that 

Vt = wt- 

That the optimal choice of s' is as given in (14 .Qp follows on observing that, with this 
choice of s', 

{d{G'G* - GG*'))'{x) = for a; > 1/, 

and 

G\y)G*{y)-G{y)&'{y) = Q. 

□ 



5. The discrete statespace case 



5.1. Additive functional case. Supppose that X is a discrete-time birth and death 
process on S* = {0, . . . , A^}, with transition matrix (P) given by 

Pn,n+i = Pn and l-pn = qn = Pn,n-i and Pn = qo = 0. 

We define 

def Qn 
Wn = — 
Pn 

and 

n 

Wn = Ylwk, 

k=l 

with the usual convention that the empty product is 1. Note that s given by 

k=0 

is the discrete scale function in that s(0) = 0, s is strictly increasing on S and s{Xt) 
is a martingale. 

Remark 5.1. Note that when we choose Pn or Wn we are actually specifying s{n + 
1) — s{n) so we shall denote this quantity by As{n) and we shall denote by ds the 
Lebesgue- Stilt jes measure on S given by 

ds{x) = As{x). 
Let / be a positive function on S and define 

fin) = ^{f{n - 1) + f{n)) for 1 < n < iV. 
Theorem 5.2. If we define 

Ty-l 

0,(x)=E.[^/(Xr)], 
t=o 

then for x < y 

(5.1) 0.(x) = fix) + ... + /(,- 1) + f £ '-^^^^^^ 

v=x u=0 " 
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while for y < x 

(5.2) = /(,+!)+...+ /(.) + £ x! 

v=y u=v+l " 

Proof: It is relatively easy to check that 4> satisfies 

H^) = PxH^ + 1) + 5^:0(2^ - 1) + f{x) 

with the right boundary conditions. □ 

It follows from this that optimal payoffs are given by essentially the same formulae as 
in the continuous case. Thus we now define the constrained control set A4£, by 



(5.3) = { scale functions s : ds\c = dso\c}- 

Remark 5.3. By convention we shall always assume that G C since we cannot 
control Wq and hence cannot control As(0). 

Theorem 5.4. For any scale function s, define the measure P by 

and the measure J by 

J{D) J2 \l 2/(^ + 1) /or D C {0, . . . , TV - 1}, 

then, given a scale function sq, 

s 

Remark 5.5. Note that all complements are taken with respect to S\ {N}. 
The optimal choice of s is given by 



As{x) ^W,^ 



W2 : onC 



/"o (C) 



2f{x + 1) :on C 



The dynamic problem translates in exactly the same way: we define the constrained 
control set: 

= {S : ds|{o,...,2/-l} = C?So|{0,...,2/-l}}, 

then we have the following. 

Theorem 5.6. For each s & A4 let denote the running maximum of the controlled 
process X^. Then for each sq & A4, the optimal payoff (or Bellman) process 
defined by 

5-1 

Vr'^ essinf E[J2f{Xndt\J^t] 



is given by 
(5.4) 
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E + 2(v/so(Mf)Po(Mf) + J(M;"))' - 0^;o(O) : /or Mf < N 



E /(Xjt^M + ■■ for Mf = N, 



( where zs formally given by equations liS. 1\) and li5.S\) with s = Sq), and the optimal 
control is to take 



so' 



(5.5) Wix) = j^^^\J~f{^ + 1) for X > Mr. 

5.2. The discounted problem. Suppose that for each x G {0, . . . , — 1}, < < 
1, then define 

== (1 — ri^iVi)"^, with r_i taken to be 1, 

def ' 

and cr(zi, 12,. . . ,ii) = Yl ^im- Now set Ak{x) = {{u,v) : < mi < f 1 < . . . < f ^ < 

m=l 

x}, Ak{x) = {{u,v) : X < Vi < Ui < . . . < Uk < N and = crmWm, where Wm is 
as before. 

Theorem 5.7. For x < y 

Ty-l 

(5.6) E^iH r^.o] =r....r,_iG(x)/G(i/), 



where 



1 -A- W^," 



k=l {u,v)eAk{x) ^-'-^m=l "™ 

w/iz/e for X > y 

Ty-l 

(5.7) i?.[n rxlA = ry+i • . . r.G(x)/G(y), 

t=o 

^ ^ n-r?/ 7)^ J- A T/T/o- 



where 



fc=l ^„ zi. ^— '— ^ni=l 



Proof- Define 4 = "^fl ^(-^D. then 

t=o 

dx = r^iPx + Qxdx-idx)- 

Setting 

rxtx/tx+i dx 

we see that 

tx+l = (1 + Wx)tx - WxTxTx-ltx-l 

or 

(^z+l ~ tx — Wxitx — tx-l = Wxi^l — rx-irx)tx~i- 

Substituting 

tx = G{x) 
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It is easy to check that this is satisfied. Now boundary conditions give equation fl5.6p . 
The proof of equation {\5.7\i is essentially the same. □ 

Now with this choice of a we get same results as before. 



Theorem 5.8. Suppose G and G are as defined in Theorem \5. 7[ we set 

k 



and 



a(u v) W" 

k=l (u,v)i^Al{x) ^— '-^m=l «™ 



^ ' ^ ^ rr^i iA J- J- T/f/o" 



where 



and 



a(u,v) W° 



A\{x) = {{u,v) : X < ui < vi < . . . < Vk < N} 



Al.{x) = {{u,v) : < f 1 < Ml < . . . < Mfe < x}. 
Then the optimal payoff' to the discrete version of Problem \1.3\ is given by 
s-i 



where 

y<xi<...<Xk<N ^ ' 
n=l 0<ui<»;i<...<-u„_i<u„<j/ \— ^ IL 

and 



AG*iy) E E 



n=l 0<ui<vi<...<v„^i<u„<y \— ' — ' £ 

Theorem 5.9. The Bellman process for the dynamic version of Problem \1.3\ is given 
by 

U rxsGiXn^iM^ : ifM^KN, 
ys ^ ) «=0 

'lirxsG^Xt): tf = N. 

ii=0 
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5.3. Continuous-time and discrete-time with waiting. Now we consider the 
cases where the birth and death process may wait in a state and also where it forms 
a continuous time Markov chain. 

By solving the problem in the generality of Theorems 15.41 to 15.91 we are are able to 
deal with these two cases very easily. First, in the discrete-time case with waiting, 
where 

Pn,n—l Qm Pn,n and Pn^n+1 Prii 

(we stress that we take the holding probabilities e„ to be fixed and not controllable) 
we can condition on the first exit time from each state — so that we replace P by P* 
given by 

p* _ „* 42^ p* _ J p* _ * 4£f (In 

n,ri.-l — 'dn ~ i _ ' n,n ~ "-"i ^'■'■'^ ra,n+l — Pn ~ i _ " 

Of course we must now modify the performance functional to allow for the time spent 
waiting in a state. Thus for the additive case we must replace / by /* given by 

/•(") = 

1 - e„ 

whilst in the multiplicative case we replace r by r* given by 

dei^ t,^ , , .,4-1 (1 - e„)r(n) 



r ' [n] = > eld — e„ Irln''*"^ 



1 — e„r(n) 

t=o ^ ' 

Then in the case of a continuous-time birth and death process with birth and death 
rates of A„ and we obtain P as the transition matrix for the corresponding jump 
chain (so Pnn-i = . and Pnn+i = t^t — • We allow for the exponential holding 
times by setting 

/•(«) = v^. 

and 



r*{n) 



a{n) + A„ + /i„ 
Thus our results are still given by Theorems I5.4H5.9I 

6. Examples and some concluding remarks 

We first consider the original time-minimisation problem with general a. 

Example 6.1. Suppose that / = 1 and we seek to solve Problem li.M Thus C 
y = and the optimal choice of s' according to Theorem \3.2\ is 

s' = — 
a 

Notice that it follows that (with this choice of scale function) 

ds{Xl) = V2dBt, 

and 

Eo[S] = silf. 
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Example 6.2. If we extend the previous example by assuming that s is given on [0, y), 
then we will still have s' proportional to - on [y, 1] and so on this interval s{X^)will 
behave like a multiple of Brownian Motion with partial reflection aty (at least if s'{y~) 
exists). 

Example 6.3. We now consider the additive functional case with general f . Then 
from Theorem \3.2l the optimal choice of s' is With this choice of s, we see that 

<s{x^) >t= f f{x:)du, 

Jo 

so that ^ 

Eo[[ f{X:)du = E[<s{X')>s]. 
Jo 

Example 6.4. If we turn now to the discounted case and take a constant and a = 1, 
we see that the optimal choice of s' is constant, corresponding to zero drift. Thus we 
obtain the same optimal control for each a. This suggests that possibly, the optimum 
is actually a stochastic minimum for the shuttle time. Whilst we cannot contradict 
this for initial position 0, the corresponding statement for a general starting position 
is false. 

To see this let sq correspond to drift 1 on [0, y\ . Then a simple calculation shows that 
the optimal choice of s' on \v,V\ is \/2a\ — '^^1^°' ■ S^ptttt^I - It is clear 

^ •' Li" J V y cosh(Vl+2ay)- ^j^^^a S'"h(Vl+2a^;) 

that this choice depends on a and hence there cannot be a stochastic minimum since, 
were one to exist, it would achieve the minimum in each discounted problem. 
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